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The  Impact  of  Conducting  ATAM®  Evaluations  on 
Army  Programs 


Objective  of  This  Study 

The  Army  Strategic  Software  Improvement  Program  (ASSIP)  is  a 
multiyear  effort  targeted  at  improving  the  way  in  which  the  Army 
acquires  software-intensive  systems. 

As  part  of  the  ASSIP,  the  Army  funded  the  Carnegie  Mellon®  Software 
Engineering  Institute  (SEI)  to  conduct  software  architecture  evaluations 
using  the  SEI  Architecture  Tradeoff  Analysis  Method®  (ATAM®).  When  a 
system’s  architecture  did  not  exist  or  was  not  ready  to  evaluate,  the 
Army  sponsored  SEI  Quality  Attribute  Workshops  (QAWs).  Other  Army 
programs  funded  their  own  ATAM  evaluations  and  QAWs. 

The  objective  of  this  study1  was  to  determine  the  value  the  Army 
programs  received  from  using  the  ATAM  and  the  QAW. 

This  impact  data  will  enable  the  Army  to  decide  whether  these  practices 
should  be  considered  for  broad  adoption  across  the  Army. 

1.  Nord,  R.L.,  Bergey,  J.,  Blanchette,  Jr.,  S.  Klein,  M.  Impact  of  Army  Architecture  Evaluations  (CMU/SEI-2009-SR-007). 
Software  Engineering  Institute,  Carnegie  Mellon  University,  2009. 
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Army  Programs 


12  Army  programs  conducted  11  ATAMs  and  5  QAWs  from  2002  through  2007. 


Army  Programs  (in  alphabetical  order) 

ATAM 

QAW 

Aerial  Common  Sensor  (ACS) 

/ 

/ 

Army  Battle  Command  System  (ABCS) 

/ 

Command  Post  of  the  Future  (CPoF) 

/ 

Common  Avionics  Architecture  System  (CAAS) 

/ 

Distributed  Common  Ground  Station  -  Army  (DCGS-A) 

/ 

/ 

Force  XXI  Battle  Command,  Brigade-and-Below  (FBCB2) 

/ 

Future  Combat  Systems  (FCS) 

/ 

/ 

Integrated  Fired  Control  (IFC) 

/ 

/ 

Joint  Tactical  Common  Operational  Picture  Workstation  (JTCW) 

/ 

Manned/Unmanned  Common  Architecture  Program  (MCAP) 

/ 

One  Semi-Automated  Forces  (OneSAF) 

/ 

Warfighter  Information  Network  -  Tactical  (WIN-T) 

/ 
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ATAM  and  QAW 


Non-Risks 


Barbacci,  M.,  Ellison,  R.,  Lattanze,A., 
Stafford,  J.,  Weinstock,  C.,  &  Wood,  W. 
Quality  Attribute  Workshops,  3rd  Edition 
(CMU/SEI-2003-TR-01 6,  ADA41 8428). 
Software  Engineering  Institute,  Carnegie 
Mellon  University,  2003. 

Clements,  R,  Kazman,  R.,  &  Klein,  M. 
Evaluating  Software  Architectures: 
Methods  and  Case  Studies.  Addison- 
Wesley,  2002. 


ATAM  Conceptual  Flow 


The  purpose  of  the  A  TAM  is  to  assess  the  consequences  of 
architectural  decisions  in  light  of  quality  attribute  requirements  and 
business  goals.  The  ATAM  helps  stakeholders  ask  the  right  questions 
to  discover  architectural  risks. 


The  QAI/l/is  a  facilitated  method  that  engages  system  stakeholders 
early  in  the  life  cycle  to  discover  the  driving  quality  attributes  of  a 
software-intensive  system. 
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Assessing  Impact 


Measurement 

Category 

Comments 

Inputs/Indicators 

Measures  inputs  into  meetings  including  the  number  of 
meetings,  attendees,  audience,  costs,  and  efficiencies 

Reaction  and 
Perceived  Value 

Measures  reaction  to,  and  satisfaction  with,  the 
experience,  ambiance,  contents,  and  value  of  meeting 

Learning 

Measures  what  participants  learned  in  the  meeting- 
information,  knowledge,  skills,  and  contacts  (takeaways) 

Application  and 
Implementation 

Measures  progress  after  the  meeting-the  use  of 
information,  knowledge,  skills,  and  contacts 

Impact  and 
Consequences 

Measures  changes  in  business  impact  variables  such  as 
output,  quality,  time,  and  cost-linked  to  the  meeting 

ROI 

Compares  the  monetary  benefits  of  the  business  impact 
measures  to  the  costs  of  the  meeting 

Phillips,  J.J.;  Breining,  M.T.;  Phillips,  P.P.  Return  on  Investment  in  Meetings  &  Events.  Elsevier,  2008. 
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Criteria  for  Evaluating  Impact  of  ATAM/QAW 

Quality 

Improve 

merits 

ATAM/QAW  Preparation  and  Execution 

ATAM/QAW  Post-Engagement  Activities 

Programmatic 

(Cost  and 
Schedule) 

Effective  in  terms  of 

•  cost 

•  effort 

Improved 

•  program  schedule  performance 

•  program  cost  performance 

Product 

(System  and 
Software) 

Clarification,  discovery,  and  use  of 

•  quality  attribute  requirements 

•  architecture  documentation 

•  risks  and  risk  themes 

Improved 

•  software  architecture 

•  system  qualities/capabilities 

•  warfighter  effectiveness 

Practices 

(Acquisition  and 
Development) 

Foster  communication  among  program  office, 
suppliers,  and  stakeholders  to 

•  understand  and  control  cost  and  schedule 

•  communicate  quality  attribute  requirements 

•  evaluate  the  architecture 

•  improve  the  architecture 

Organizational  changes 

•  use  of  the  results 

•  use  of  the  practices  in  the  short  term 

•  adoption  of  the  concepts 

•  adoption  of  the  methods 

•  training  personnel  to  conduct  the  methods 
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Questionnaire 

The  questionnaire  was  organized  into  four  sections: 

1.  Conducting  the  ATAM/QAW -  elicited  information  about  product  and 
practice  improvements  during  preparation  and  execution  of  the 
method. 

2.  Follow-On  ATAM/QAW  Activities  -  elicited  information  about  practice 
improvements  during  the  post  activities,  focusing  on  how  the 
engagement  affected  the  immediate  behavior  of  the  organization. 

3.  Adoption  of  ATAM/QAW  -  elicited  information  about  practice 
improvements  during  the  post  activities,  focusing  on  how  the 
engagement  affected  the  long-term  acquisition  practices. 

4.  Overall  Impact  -  elicited  information  about  short-term  and  long-term 
programmatic  improvements  and  long-term  product  improvements;  in 
addition,  it  provided  survey  respondents  an  opportunity  to  share 
comments  on  how  they  perceived  the  overall  impact. 
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Factors  Affecting  Impact 

Undesirable  Context  /I  ^  , 
(#  programs)  - 

Desirable  Context 
- (#  programs) 

8  Planning 

1 

5  Timing 

4 

1 

Need  6 

3 

Motivation  7 

4  Accomodation 

3 

2 

Follow  Through  5 

Context  of  use  for  each  factor  was  rate 
1)  undesirable,  2)  somewhat  undesiral 

5d  on  a  three  point  scale: 
ble/desirable,  3)  desirable. 
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Overview  of  Results  - 1 

Overall,  the  survey  results  suggest  that  the  Army  programs  received 
benefit  from  the  use  of  the  ATAM  and  QAW.  Context  of  use  had  a 
significant  impact  on  these  findings,  leading  us  to  believe  that  under 
appropriate  acquisition  conditions  these  practices  are  very  likely  to  have 
a  positive  impact  on  system  quality. 

•  Six  of  the  12  programs  reported  that  it  cost  less  to  use  the  ATAM/ 
QAW  than  the  techniques  they  traditionally  have  used.  Moreover, 
they  all  reported  results  that  were  at  least  as  good,  and  often  better, 
than  the  results  they  traditionally  obtained. 

•  Ten  of  the  12  programs  reported  that  the  ATAM/QAW  provided  an 
informed  basis  for  the  program  office  and  the  supplier  to  better 
understand  and  control  the  software  development  cost  and  schedule. 
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Overview  of  Results  -  2 

•  All  programs  found  that  using  the  ATAM  /QAW  increased  their 
understanding  of  the  system’s  quality  attribute  requirements,  design 
decisions,  and  risks. 

•  Overall,  the  programs  felt  that  the  use  of  the  ATAM/QAW  provided  a 
good  mechanism  for  the  program  office,  suppliers,  and  stakeholders 
to  communicate  their  needs  and  understand  how  they  are  met. 

•  A  majority  of  the  respondents  felt  that  using  the  ATAM/QAW  led  to 
an  improved  architecture  (8  of  12),  and  a  higher  quality  system  (6  of 
10).  A  minority  of  the  respondents  felt  that  using  the  ATAM/QAW 
would  result  in  overall  cost  and  schedule  reductions  for  their 
respective  programs. 
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Programmatic  Improvements 


Conducting  the  method 

Better  (or  same) 

Lesser 

Cheaper  (or  same) 

6 

1 

Costlier 

4 

0 

Quality  of  ATAM/QAW  Outputs  vs.  Other  Techniques 
(shown  as  number  of  programs) 


Overall,  the  data  shows  that  the  ATAM  and  QAW  are  effective  techniques 
for  eliciting  quality  attribute  requirements  and  analyzing  software 
architecture;  in  some  cases,  they  are  more  cost-effective  than  traditional 
analysis  methods. 

In  the  other  cases,  people  reported  context  of  use  affected  their  response 
and  that  they  would  use  the  methods  in  the  future 

•  “QA  W  and  A  TAM  provided  benefits  deemed  substantial  enough  to  warrant 
adoption  for  future  contracts.  ” 
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Product  Improvements 


Minimal  Moderate  Significant  Very  Substantial 

■  Quality  Attributes  ■  Architecture  ■  Risks 


Architecturally  Significant  Artifacts  Enhanced  byATAM/QAW 
(total  number  of  programs  is  12) 


These  results  demonstrate  that  the  architecture  team  is  able  to  use 
ATAM/QAW  to  achieve  an  understanding  of  stakeholder  expectations 
for  the  system,  the  implications  of  architectural  decisions  on  user  needs, 
and  the  relevant  risks  to  success. 
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Practice  Improvements 

12 


The  significance  of  these  results  is  that  stakeholders,  collectively,  are 
able  to  use  ATAM/QAW  to  achieve  a  common  understanding  of  the 
system  under  development,  making  it  more  likely  that  the  completed 
product  will  address  stakeholder  expectations  and  user  needs,  thereby 
improving  chances  for  program  success. 
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Transition 


1 .  All  programs  reported  some  use  of  the  artifacts  produced  by  the 
ATAM/QAW.  For  example,  some  put  the  quality  attribute  scenarios 
they  developed  into  a  requirements  tracking  system,  others  improved 
their  architecture  documentation,  and  others  formally  tracked  risks 
discovered  during  the  evaluation. 

2.  Eleven  programs  reported  using  the  techniques  of  the  ATAM / 
QAW  to  uncover  additional  risks  by,  for  example,  refining  or 
analyzing  additional  scenarios. 

3.  Nine  programs  reported  adopting  the  concepts  of  quality  attribute 
requirements  elicitation  and  architecture  evaluation. 

4.  Seven  programs  reported  adopting  the  ATAM/QAW  methods 

(i.e.,  by  using  or  specifying  the  use  of  the  practices). 

5.  Three  programs  reported  investment  in  formal  ATAM  training. 
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Conclusion 


The  data  gathered  for  this  study  confirms  that  the  use  of  ATAM 
evaluations  and  QAWs  are  generally  beneficial  to  DoD  system 
acquisitions  and  suggests  that  maximal  benefit  is  achievable  only  if 
architecture-centric  practices  are  pre-planned  and  built  into  the 
acquisition  process. 


-  ^  „  Impact  of  Conducting  ATAM  Evaluations 

Software  Engineering  Institute  Carnegie  Mellon  Nord,  May  2009 

©  2009  Carnegie  Mellon  University 


4/27/09 


Software  Engineering  Institute 


Carnegie  Mellon 


©  2009  Carnegie  Mellon  University 


8 


The  Impact  of  Conducting  ATAM®  Evaluations  on 
Army  Programs 


System  Development  and  Demonstration  Phase 

-  Proactive  Adoption  of  an  Architecture-Centric  Approach  - 


Must 


System  Development  and 
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